Skip to content

feat(sdk): canonical-object pipeline — SchemaHandler table, BufferAnchor, push_message_v2 ABI#86

Open
pabloinigoblasco wants to merge 5 commits into
mainfrom
feat/canonical-object-pipeline
Open

feat(sdk): canonical-object pipeline — SchemaHandler table, BufferAnchor, push_message_v2 ABI#86
pabloinigoblasco wants to merge 5 commits into
mainfrom
feat/canonical-object-pipeline

Conversation

@pabloinigoblasco
Copy link
Copy Markdown
Collaborator

Summary

A coherent set of changes across pj_base and pj_plugins that establishes the canonical-object pipeline — the contract by which MessageParser plugins produce decoded scene primitives (images, point clouds, …) and the host routes them through the ObjectStore.

This is a design sketch posted as a draft for review. It compiles cleanly with the companion parser_ros / runtime work and unit tests pass, but the end-to-end flow against real data sources is still pending validation.

What this adds

Canonical-object SDK (pj_base/sdk/canonical_object.hpp)

  • BufferAnchor + PayloadView: zero-copy payload sharing — non-owning Span<const uint8_t> plus a shared_ptr<const void> that keeps the underlying bytes alive.
  • sdk::Image, CompressedImage, PointCloud built around view + anchor; parsers return them without copying source bytes.
  • CanonicalObjectKind enum + SchemaClassification.
  • PixelFormat covers RGB/RGBA, BGR/BGRA, Mono8/Mono16.

MessageParser plugin base

  • SchemaHandler table: per-schema registration with parse_scalars and parse_object callables. The base's classifySchema / parseScalars / parseObject are now table lookups. Plugins call registerSchemaHandler(...) in their constructor or bindSchema to declare what they know.
  • parse() is no longer pure virtual: default implementation routes through parseScalars + writeHost.appendRecord, so plugins that register all their schemas inherit parse() for free.

C ABI extension

  • PJ_payload_t / PJ_payload_anchor_t / PJ_payload_fetcher_t cross-ABI types: idempotent byte-fetcher with a release callback.
  • New push_message_v2 tail slot on PJ_data_source_runtime_host_vtable_t: the DataSource hands the host an idempotent fetcher; the host applies the active ObjectIngestPolicy (kPureLazy / kLazyObjectsEagerScalars / kEager) to decide when (and whether) to invoke it.
  • abi_layout_sentinels_test vtable size updated deliberately: 80 → 104 (MIN_VTABLE_SIZE pinned at the v4.0 baseline of 80).

SDK C++ helpers

  • DataSourceRuntimeHostView::pushMessage template wraps a C++ closure (returning PayloadView or vector<uint8_t>) into a PJ_payload_fetcher_t and delegates to push_message_v2. Returns an explicit error when the host doesn't expose the slot — no silent fallback to the legacy raw-message path.
  • ObjectIngestPolicy + ObjectIngestPolicyResolver with hierarchical override cascade: topic > data_source > kind > default.
  • MessageParserHandle::classifySchema wrapper for the tail-slot call.

Tests

  • object_ingest_policy_test: cascade rules at all four levels + last-write-wins.
  • push_message_v2_test: mock host exercising the template's fetcher wrap (vector + PayloadView shapes), idempotency under repeated fetch, ctx lifetime via shared_ptr canary, anchor propagation past fetcher release, and the explicit error when the host predates the slot.

Test plan

  • Build clean (RelWithDebInfo).
  • Unit tests pass (object_ingest_policy_test, push_message_v2_test, abi_layout_sentinels_test).
  • End-to-end with the companion parser_ros / runtime-host work — pending.

Companion work

The parser-side (parser_ros), the file source (mcap_source) and the runtime host (in the consumer app) ship as separate PRs to keep review surfaces focused.

…hor, push_message_v2 ABI

A coherent set of changes across pj_base and pj_plugins that establishes
the canonical-object pipeline.

Canonical-object SDK (pj_base/sdk/canonical_object.hpp):
- BufferAnchor + PayloadView for zero-copy payload sharing (Span<const
  uint8_t> view + shared_ptr<const void> anchor).
- sdk::Image, CompressedImage, PointCloud canonical types built around
  the view+anchor pattern so parsers can return them without copying
  the source bytes.
- CanonicalObjectKind enum + SchemaClassification descriptor.
- PixelFormat with kRGB888/kRGBA8888/kBGR888/kBGRA8888/kMono8/kMono16.

MessageParser plugin base:
- SchemaHandler table: per-schema registration with parse_scalars and
  parse_object callables. The base's classifySchema / parseScalars /
  parseObject methods are now table lookups. Plugins call
  registerSchemaHandler() in their constructor (or in bindSchema) to
  declare what they know about each type name.
- parse() is no longer pure virtual: default implementation routes
  through parseScalars + writeHost.appendRecord, so plugins that
  register all their schemas via the table inherit parse() for free.

C ABI:
- PJ_payload_t / PJ_payload_anchor_t / PJ_payload_fetcher_t cross-ABI
  types: idempotent byte-fetcher with a release callback.
- New push_message_v2 tail slot on PJ_data_source_runtime_host_vtable_t:
  the DataSource hands the host an idempotent fetcher; the host applies
  the active ObjectIngestPolicy (kPureLazy / kLazyObjectsEagerScalars /
  kEager) to decide when (and whether) to invoke it.
- vtable size sentinel updated deliberately: 80 -> 104 bytes
  (MIN_VTABLE_SIZE pinned at the v4.0 baseline of 80).

SDK C++ helpers:
- DataSourceRuntimeHostView::pushMessage template: wraps a C++ closure
  (returning either PayloadView or vector<uint8_t>) into a
  PJ_payload_fetcher_t and delegates to push_message_v2. Returns an
  explicit error when the host doesn't expose the slot — no silent
  fallback to the legacy raw-message path.
- ObjectIngestPolicy + ObjectIngestPolicyResolver with hierarchical
  override cascade: topic > data_source > kind > default.
- MessageParserHandle::classifySchema wrapper for the tail-slot call.

Canonical-object blob serialization:
- Flat byte layout for Image / CompressedImage / PointCloud crossing
  the C ABI. Writer/reader pair under sdk/detail/.

Tests:
- object_ingest_policy_test: cascade rules at all four levels +
  last-write-wins.
- push_message_v2_test: mock host exercising the template's fetcher
  wrap (vector and PayloadView shapes), idempotency under repeated
  fetch, ctx lifetime via shared_ptr canary, anchor propagation past
  fetcher release, and the explicit error when the host predates the
  slot.

Status: design sketch posted as a draft. Compiles cleanly with the
companion parser/runtime work; not yet exercised end-to-end against
real data sources.
No behavior change. Reformats SDK headers, ABI headers, message
parser plumbing, and the new ingest policy / push_message_v2 tests
per .clang-format (Google style, 120-col, InsertBraces: true).
Output of running pre-commit's clang-format hook over the files
touched in this branch.
The file-level docstrings of canonical_object.hpp and
object_ingest_policy.hpp pointed at a private report path that
does not exist in this repository. Remove the dangling references;
the surrounding prose already explains the design.
…functional scalars

The canonical-object pipeline and the pure-functional scalar path are C++
SDK contracts: MessageParserPluginBase::parseObject() and parseScalars()
are called directly on the C++ pointer by the in-process runtime host,
preserving zero-copy via BufferAnchor. The C ABI slots (parse_object,
parse_scalars) and their wire-format support are removed; pure-C plugins
emit scalars via the parse() slot writing to writeHost.

Removed:
- detail/canonical_object_serialization.hpp (full file).
- PJ_canonical_object_blob_t and PJ_named_field_value_buffer_t from
  canonical_object_abi.h.
- parse_object and parse_scalars slots from PJ_message_parser_vtable_t.
- trampoline_parse_object and trampoline_parse_scalars.
- Per-instance buffers (scalars_owned_buf_, scalars_abi_buf_,
  object_blob_buf_) that only the trampolines used.

Kept:
- SchemaHandler::parse_scalars / parse_object (C++ callables registered
  by plugins) and MessageParserPluginBase::parseScalars / parseObject
  (C++ methods invoked by the host).
- classify_schema C ABI slot (used by MessageParserHandle::classifySchema).
- The parse() slot for pure-C plugins that emit scalars to writeHost.

Vtable size: PJ_message_parser_vtable_t shrinks from 104 to 88 bytes
(80 v4.0 baseline + 1 tail slot for classify_schema).
PJ_MESSAGE_PARSER_MIN_VTABLE_SIZE stays pinned at 80.
ObjectIngestPolicy is a host-owned concern: the host instantiates its own
ObjectIngestPolicyResolver during file-load setup and consults it on each
push_message_v2 dispatch. DataSource plugins are policy-agnostic — they
fabricate a payload fetcher via runtimeHost().pushMessage() and hand it
off without inspecting or configuring policy.

Exposing a per-view resolver in DataSourceRuntimeHostView contradicted
that contract: mutations went to a local instance the host never read.
The accessor is removed; the type sdk::ObjectIngestPolicyResolver stays
in pj_base/sdk/object_ingest_policy.hpp as host-side vocabulary.

Changes:
- Drop the objectIngestPolicy() accessor.
- Drop the mutable policy_resolver_ member.
- Replace transitive include of object_ingest_policy.hpp with an explicit
  include of canonical_object.hpp (needed for PayloadView in pushMessage).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant